54 research outputs found
Resampled Priors for Variational Autoencoders
We propose Learned Accept/Reject Sampling (LARS), a method for constructing
richer priors using rejection sampling with a learned acceptance function. This
work is motivated by recent analyses of the VAE objective, which pointed out
that commonly used simple priors can lead to underfitting. As the distribution
induced by LARS involves an intractable normalizing constant, we show how to
estimate it and its gradients efficiently. We demonstrate that LARS priors
improve VAE performance on several standard datasets both when they are learned
jointly with the rest of the model and when they are fitted to a pretrained
model. Finally, we show that LARS can be combined with existing methods for
defining flexible priors for an additional boost in performance
A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
In spite of their superior performance, neural probabilistic language models
(NPLMs) remain far less widely used than n-gram models due to their notoriously
long training times, which are measured in weeks even for moderately-sized
datasets. Training NPLMs is computationally expensive because they are
explicitly normalized, which leads to having to consider all words in the
vocabulary when computing the log-likelihood gradients.
We propose a fast and simple algorithm for training NPLMs based on
noise-contrastive estimation, a newly introduced procedure for estimating
unnormalized continuous distributions. We investigate the behaviour of the
algorithm on the Penn Treebank corpus and show that it reduces the training
times by more than an order of magnitude without affecting the quality of the
resulting models. The algorithm is also more efficient and much more stable
than importance sampling because it requires far fewer noise samples to perform
well.
We demonstrate the scalability of the proposed approach by training several
neural language models on a 47M-word corpus with a 80K-word vocabulary,
obtaining state-of-the-art results on the Microsoft Research Sentence
Completion Challenge dataset.Comment: Appears in Proceedings of the 29th International Conference on
Machine Learning (ICML 2012
Learning Item Trees for Probabilistic Modelling of Implicit Feedback
User preferences for items can be inferred from either explicit feedback,
such as item ratings, or implicit feedback, such as rental histories. Research
in collaborative filtering has concentrated on explicit feedback, resulting in
the development of accurate and scalable models. However, since explicit
feedback is often difficult to collect it is important to develop effective
models that take advantage of the more widely available implicit feedback. We
introduce a probabilistic approach to collaborative filtering with implicit
feedback based on modelling the user's item selection process. In the interests
of scalability, we restrict our attention to tree-structured distributions over
items and develop a principled and efficient algorithm for learning item trees
from data. We also identify a problem with a widely used protocol for
evaluating implicit feedback models and propose a way of addressing it using a
small quantity of explicit feedback data.Comment: 8 page
- …